Roadmapping.
Commenting on some prioritisation issues. Trying to formulate a metadata roadmap for the CADC tech stack (alongside DaCHS).
Debugging Susana's issues (checksum overhead before upload, timeouts on upload to STFC_STORM) Meeting in afternoon (data management prioritisation)
Looking at websockets.
Checked over Manu's transcript. Some debugging with Susana.
Finished looking over annual review document & career matrix.
Looking over annual review document & career matrix.
Demo, annual review, meetings.
Finishing IVOA stories (prepping IVOA demo for SRC ART)
Finishing IVOA stories (prepping IVOA demo for SRC ART)
Preparing SIG metadata presentation for Rucio workshop. Rucio workshop organisation.
Preparing SIG metadata presentation for Rucio workshop. Rucio workshop organisation.
Preparing SIG metadata presentation for Rucio workshop. Rucio workshop organisation.
Rucio workshop organisation.
Demoed IVOA work for Dave M:
Outline:
Demonstrate what's been done:
- Browser (repo) gloss over service architecture
- Docker up (docker-compose -f docker-compose.yml -f docker-compose.sandbox.yml up)
- Database endpoint (email@email.com, secret)
- Dummy rucio table (refer to postgres_meta plugin, show structure as json blob)
- IVOA trigger (on upsert into the dids table, this function explodes keys in the json blob in to columns in to another table, the ObsCore table), and show ObsCore table
- This is essentially the workflow that syncs Rucio managed data into the format required by ObsCore
- Jump to TAP (format: HTML, select * from ivoa.ObsCore) - (have a web interface allowing me to query that obscore table)
- Jump to Jupyter and do same command (admin, secret)
- Architecture discussion, go to considerations section
- Set up with a trigger currently: data coming into the rucio dids table is synced with the obscore table. This duplicates data but allows physical clustering and table partitioning
- Could do it with a view, removes duplication but doesn't allow for clustering
- Could use one table for both Rucio/IVOA (back to slides)
- Currently have objective complete (show KR)
- Feature is worded such that this toy model is deployed on a Rucio production instance, and then to pump in some mock metadata
- Is it more worthwhile to expand this model first to include SIA protocol for example
Discussion:
- does work, triggers OK, but scale? best to get postgres mixed metadata plugin working
- http get SIA (endpoint minimal set of params, gives response) -> implementations out of box e.g. python DaCHs, uses table. Using is fairly straightforward, XML, input "grammar"
- http://docs.g-vo.org/DaCHS/ <— DaCHS documentationIVOA related stories (prepping a demo for Dave M.)
Career matrix, rucio workshop organisation